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Verification of Images of Persons 

Field of Invention 

5 

The invention relates to a method for verifying images of persons, in particular, portraits. 
Predetermined quality standards for an image of a person (a portrait) are parameterized 
by threshold values and by comparison of the detected characteristic values of the image 
with the threshold values it is verified whether the quality standards are observed during 
10 performance of the method. The method allows for objectively verifying according to pre- 
determined criteria whether an image of a person is appropriate for person identification. 
Furthermore, there are provided apparatuses for performing the method. 

15 Prior Art 



The oldest and still most prevalent method for recognizing or verifying respectively, the 
identity of a person is face recognition. 

20 In difference to the comparison of fingerprints or signatures or a genetic comparison ("ge- 
netic fingerprint") the visual comparison employed in face recognition can be readily and 
quickly be performed by the unskilled person without any technical means. 

Identity verification by face recognition is always performed by a comparison of two im- 
25 ages: (1) a reference image that is either available as a real portrait, e.g., as a photo on 
an identity card, a search photo or as a different portrait, or as a physical or imaginary 
image reconstructed from the memory or a description, e.g., a photofit picture, with (2) a 
comparative Image to be compared with the reference image, whereupon the identity is 
asserted or not depending on the grade of correspondence. 

30 

Such kinds of identity verification are usually still performed by persons rather than being 
executed automatically, as, e.g. in a traffic check or passport control in which cases the 
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reference image is a photo on an identity card and the comparative image is an imaginary 
image reconstructed by the controlling person from the visual impression of the face of 
the checked person. Methods of automatic image processing are very laborious due to 
the huge amount of image information to be processed and also error-prone, if the qual- 
.5 ity of the used images is insufficient. Features that are not considered as quality features 
may also play some role. For example, the posture of the head has a huge impact on the 
significance of the outcome of an automatic comparison of images: according to the art 
front views qualify much better than other postures as, e.g.. the half-side profile of a per- 
son's face. 

10 

Either identity verification is performed visually be a person or automatically, the quality of 
the used portraits is a crucial criterion for the significance of an identity verification. For 
the quality not only the reproduction qualities as the resolution, sharpness, contrast, 
brightness, staining etc. of the image but also the mode of presentation of the displayed 
15 face is of importance. Particularly, for automatic methods portraits of reduced quality rep- 
resent almost unsolvable problems in successfully comparing images. 

Identity verification of persons by comparison of fingerprints, signatures, etc. is to an in- 
creasing degree peri^omned with the assistance of semi-automatic or automatic methods. 

20 Again, the quality of the used fingerprints or signatures, respectively. Is crucial. With re- 
spect to signatures the US patent 6,363,162 A. e.g., describes a system and method for 
verifying the quality of signatures wherein a binary image of the document carrying the 
signature is generated, the background is eliminated and subsequently the signature is 
examined and evaluated. However, this method is of no or only very limited use for the 

25 application to portraits due to the higher complexity of such images. 

In fact, methods and systems for automatically verifying and. if need be, correcting 
the contrast, color and brightness of images are known in typography and photography. 
These are, however, always based on an analysis of either the entire image or predeter- 
30 mined image areas. They are not appropriate for the extraction of structures or even iden- 
tification of individual objects within the image. 
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A further problem lies in the fact that quality standards for portraits are only vaguely de- 
fined and up to now are only verified by visual inspection. The general appearance of a 
German passport photograph, e.g., is defined by some criteria and examples. However, 
whether a photograph is acceptable for a passport or not is decided by one or more ex- 
5 ecutives/appointees. 

A more precise definition of quality standards is, however, meaningful only, if observation 
thereof can be verified objectively. An objective inspection method for portraits including 
verification whether an image actually observes predetermined criteria or not is, however, 
10 not available as yet. 

Description of the invention 

15 It is an object of the invention to overcome the above-described disadvantages and un- 
certainties in quality verification of images of persons and, in particular, portraits. 

This object is solved according to the invention by providing methods and apparatuses or 
systems, respectively, by means of which predetermined quality standards (quality crite- 
20 ria) for images of persons can objectively be verified. 

This object is specifically solved by the method according to claim 1 , the data processing 
means according to claim 13 or 15, respectively, and the system according to claim 16. 

25 Another aspect of the invention is to modify, if possible, images that do not observed the 
criteria such that the respective modified images of persons observe the criteria. For this 
purpose, there are provided the method described in claim 12, the data processing 
means according to claim 14 or 15, respectively, and the system according to claim 16. 

30 Advantageous embodiments of the Invention are given in the dependent claims. 
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The method for verification of digitized images of persons according to the invention 
comprises the steps of: segmenting the image of a person in a background area and a 
head or face area, analyzing the head or face area to detect at least one characteristic 
value, comparing the at least one characteristic value with at least one predetermined 
5 threshold value. 

The threshold values represent quality standards, quality criteria, and depending on the 
kind of criterion a respective characteristic value is determined to observe the respective 
quality criterion, if it is larger than the corresponding threshold, smaller than the corre- 
10 spending threshold or lies between a corresponding lower threshold and a corresponding 
upper threshold. 

With respect to the resolution and sharpness of the image, e.g., lower threshold values 
are appropriate, with respect to the size (amount of data) and brightness of the image 
15 lower and upper threshold values are appropriate, respectively, and with respect to stain- 
ing or other defects of the image (e.g., locally damaged or missing image data) upper 
threshold values are appropriate. 

Images of persons include, e.g., a live image recorded by a camera that has to fulfill the 
20 same quality standards for identification methods as a passport photograph etc. and is to 
be verified with respect to these quality standards. If the quality of the live image does not 
observe the quality standards the recording process is to be stopped or repeated, respec- 
tively. A typical application is, e.g., the automatic comparison of images in passport con- 
trol during which the live recording of the person is compared with the photograph on the 
25 passport. 

Since the hairs of a person represent one of the features that are manipulated most eas- 
ily, In many cases it is appropriate to not use the entire displayed head but rather the face 
(or at least placing emphasis on the face rather than the hairs) In Identification of persons 
30 by comparison of Images. The ears are considered as being part of the face. In this case, 
the Image of a person or the portrait, respectively, can be segmented directly in a face 
area and a background area (residual part of the image). 
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A different weighting of different regions of the head can be achieved by segmenting the 
image of a person or the portrait, respectively, in a background area and a head area 
and, in the simplest case, by assigning different weight factors to different regions of the 
5 head. In many cases, complementary, non-linear weighting of these regions is preferred. 

One of the simplest segmentation methods is based on discrimination by color and a pre- 
determined color spectrum for the face area and/or the background area is assumed. 
Particularly, assuming a predetermined color spectrum for the face area is in many cases 
10 suitable, since, e.g., passport photographs are usually taken in front of a monochrome 
background. The segmentation step may be performed by at first segmenting the image 
of a person or the portrait, respectively, and by subsequently further segmenting the head 
area in a face region and the region around the face (e.g., including hairs, the cervix etc.). 

15 Another simple segmentation method is based on the fact that images of a person or por- 
traits, respectively, in most cases, show a diffuse background or at least a background 
that is much less sharper than the head/face area. In comparison to the sharper image 
areas the blurred area is characterized by a lower local brightness variation and less 
abrupt color changes. 

20 

Complementary or in addition to these simple segmentation methods, an object recogni- 
tion method may be employed in segmentation, e.g., the graph matching method that is 
well-known in the art (see the German patent 44 06 020, Verfahren zur automatisierten 
Erkennung von Objekten) or advanced methods as the hierarchic image model matching 

25 HGM (see the European patent application No. 01 1 18 536.0). Other object recognition 
methods, e.g., are based on an eigenvalue ansatz as the method known as the eigen- 
faces method (M. Turk. A. Pentland: "Eigenfaces for recognition". Journal of Cognitive 
Neuroscience, 3(1), pp. 71-86, 1991) or are based on the analysis of local object features 
("Local Feature Analysis". J.S. Penev, J.J. Atick, Local feature analysis: A general statis- 

30 tical theory for object representation, Network: Computation in neural systems, 7, pp. 
477-500). 
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In a preferred embodiment of the method according to the invention analyzing the head or 
face area comprises detecting at least one geometric feature of the head or face area. 
'Geometric feature' is to be understood in the broadest meaning: Geometric features 
comprise, e.g., the portion of the area of the entire image covered by the head/face area, 
5 the position or spatial orientation of the head or face area within the image, the shape of 
the head/face, the width/height of the head/face area measured by the number of pixels, 
etc. It has been determined empirically that at least a resolution of 100 pixels in the hori- 
zontal direction and of 125 pixels in the vertical direction is required for a face to be veri- 
fied reasonably in the identification of a person by some method of comparing images. 

10 

In another preferred embodiment of the method according to the invention analyzing the 
head/face area comprises detecting at least one reproduction feature of the head/face 
area. These reproduction features, which are heavily influenced by the conditions of tak- 
ing the image, include contrast, color distribution, sharpness of the image, brightness, 
15 irradiation, etc. Another reproduction feature is the "grade of staining" that is of particular 
importance, if the digitized portrait or image of a person, respectively, has been repro- 
duced from a stained or damaged original copy. 

In a particularly preferred and advantageous embodiment analyzing the head or face area 
20 comprises localizing of sub-regions (e.g., eyes, mouth, nose) of the head or face area. 
Localizing or extracting of such sub-regions may, e.g., be perfomried by segmentation 
methods. At this it can be detected, amongst others, which parts of the face are displayed 
in the image (or recognizable) and which parts are not, whether eyes or mouth are closed 
or open, whether pigments or scars are present, whether eyeglasses or face piercing or 
25 earrings are present etc. Preferably, the localized search regions are not only detected, 
i.e. presence or absence is determined, but also separately examined. One feature that 
can be detected and that is of particular importance is the eye color. However, other fea- 
tures of the eyes, the nose, the mouth, in particular, the shapes thereof can be examined 
and characteristic values for the shapes, colors etc. can be determined. 

30 

According to an advantageous embodiment of the above mentioned methods as a further 
step - preferably as the first step - the image of a person/portrait is examined with respect 
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to criteria relevant for the entire image and the corresponding characteristic values are 
detemiined. These global criteria include: size, resolution, gray values, brightness, con- 
trast, color distribution, color balance, texture etc. Furthermore, subsequent to the seg- 
mentation an analysis of not only the foreground (i.e. the head or face area) but also of 
5 the background can be carried out. Again features as the portion of the area covered, 
resolution, brightness, contrast, color, color distribution, color balance, texture etc. can be 
examined. 

If at least one of the quality standards or a predetemiined number of criteria and/or a 
10 combination thereof is not fulfilled during the above described methods by an image of a 
person/portrait, it is determined that the image does not observe the predetermined qual- 
ity standards and therefore is not suitable for person recognition and, in particular, auto- 
matic person recognition. 

The quality standards can be classified in "soft" and "hard" criteria. "Hard" criteria cannot 
be amended without arbitrary assumptions. For example, red eyes that often are present 
in photographs taken with the help of flash lights usually cannot be turned into green, blue 
etc. without any further infomriation. If, however, further information is available, such a 
correction might be tolerable. In the case of "soft" criteria image corrections can be done 
based on reasonable assumptions such that the corresponding corrected image observes 
the predetermined quality standards. These criteria include, e.g., color and where appro- 
priate brightness and contrast also. Although a high image resolution, in general, is de- 
sired a too high resolution might slow the above-described methods for image processing 
down in a way that image correction including a reduction of the resolution without crucial 
loss of information can be advantageous. 

As explained above under certain circumstances it might be possible to modify an image 
that was negatively evaluated in the first instance by image correction means such that 
the modified image observes the quality standards. Therefore, the method comprises 
30 according to a preferred embodiment a step in that it is detemnined whether a negatively 
evaluated image could fulfill the predetemnined quality criteria after image correction. 



7 



In practice, this method can be carried out by determining whether the negatively evalu- 
ated image would not observe the predetermined quality standards even after having 
been subject to image correction. If the result of this determination is negative, it is as- 
sumed that after performing image correction the image might observe the predetermined 
5 quality standards. 

As mentioned at several places the invention furthermore provides a correction method 
for digitized images of persons and, in particular, portraits, and particularly, for such im- 
ages by that at least one quality criterion during one of the verification methods according 
10 to the invention is not fulfilled, wherein the correction method comprises one of or in suit- 
able combination at least some of the following steps: correcting at least one image re- 
production feature in the entire image area / head or face area / background area, and 
correction of at least one geometric feature of a head or face area displayed in the image 
of a person or the portrait. 

15 

The invention furthermore provides a data processing means configured to execute the 
inventive verification method according to claim 1 or the correction method according to 
claim 12, respectively, or to execute both methods. The data processing means may be, 
e.g., a CPU that reads digitized image data from a memory, processes the image data 
20 and outputs the processed image data or the results of the verification or correction 
methods, respectively. 

Furthermore, the invention provides a system for quality verification and correction of im- 
ages of persons comprising a storage means for storing digitized image data and at least 
25 one of the data processing means according to the invention. 

The means for storing digitized image data can be provided locally at the same place as 
the data processing means or separated from the latter, e.g., as a remote data base with 
read option and possibly write option allowing access via the Internet. 

30 

The data to be examined or reference image data or both kinds of image data can be 
stored in the storage means. 



8 



In a preferred embodiment the system according to the invention furthermore comprises 
an apparatus for generating digitized image data, e.g., a scanner or a digital camera. The 
generated image data can be stored in the above-described storage means and can be 
5 examined with respect to their features (quality) concurrently or when desired. 

In another preferred embodiment the system comprises a display device for displaying 
whether a verified image or a corrected image observes predetermined quality standards. 

10 In the following the nature of the invention and preferred methods for performing the 
method according to the invention will be described with reference to the accompanying 
drawing: 

Fig. 1 is a schematic flow chart of a particularly preferred embodiment of the method ac- 
15 cording to the invention. 

The flow chart shown in Fig. 1 represents the operation scheme of a quality verification of 
an image of a person, in this case, of a portrait, that is perfomned according to the princi- 
ples of the present invention. 

20 

In this flow chart the reference sign 1 indicates digitization/transmission of an image, the 
reference sign 2 indicates verification of an image with respect to global criteria (i.e. crite- 
ria that apply to the entire image), the reference sign 3 indicates segmentation (locating 
the head/face) and verification of criteria in the head/face area and where appropriate in 
25 the background area also, the reference sign 4 indicates verification of local criteria in the 
face area, the reference sign 5 indicates (optional) methods for image correction and the 
reference sign 6 Indicates signalizing of the results and/or the data export. 

It should be noted that the sequence is shown in the Figure for example and not all of the 
30 details are mandatory. For example, some of the correction methods can be already per- 
formed prior to block 5. 
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The digitization/transmission of images indicated in block 1 comprises generation of a 
digitized image (if need be) and reading of a digitized image in a computational unit that 
executes the steps of the subsequent blocks 2 - 6. 

5 If the image to be examined (image data) is not available in digitized form, it can be con- 
verted to a digitized imaged by means of a scanner. If a particular image format is desired 
for the perfomiance of the method, prior to or subsequent to the read operation some 
format conversion can be carried out. Common formats for digital image data are, e.g., 
JPG. JPEG. TIFF. BMP, that are well-known in the art and have not to be described in 
10 detail. It goes without saying that transmission of the image data can be carried out via 
the Internet. 

In step 2 the read image is examined with respect to global criteria. 
15 Examination includes: 

- verification of the compression factor in the case of electronically transmitted im- 
ages 

- verification of the size/resolution of the image 
20 - verification of the sharpness of the image 

- verification (and if need be correction) of the global brightness of the Image 

- verification of (and if need be correction) of a gray value or a illumination gradient, 
respectively 

- verification (and if need be correction) of the contrast of the image 
25 - verification of the of the highlight areas of the image 

- verification of the color distribution (and if need be correction of the color balance) 

- verification whether staining effects or other perturbations (e.g., stamps) are pre- 
sent in the image 

- (gamma correction) 

30 



With respect to the steps above set in brackets it is stressed that the image correction 
specified in block 5 with respect to the global features of the image reproduction may ad- 
vantageously be performed in block 2 where appropriate. 

5 In the following, these individual steps will be described in more detail. 

The compression factor, e.g., can be detected from the file description in the case of elec- 
tronically transmitted JPEG-lmages. The thus detected compression factor is compared 
with predetermined thresholds. If it falls below the threshold, a negative verification result 
10 is obtained. In face recognition, e.g., compression factors above 12.5 to 1 have been 
proven to be problematic for JPEG-based methods. 

Then, the portrait that is digitized by means of a scanner or that is already available as a 
digital image is analyzed with respect to the resolution of the image. The resolutions of 
15 the image in the x- and y-directions are compared with predetermined thresholds. In the 
case of person recognition, a minimum resolution of 100 pixels (x-direction, width of the 
image) and 125 pixels (y-direction, height of the image) have been approved being suffi- 
cient for the head/face area. 

20 The sharpness of the image is preferably judged by the high frequency spectral contribu- 
tions to the image. An estimator for the noise process may be employed, additionally, to 
discriminate between signal and noise that also contributes to high frequency portions 
and thus, the noise spectrum can be subtracted. The spectral portions that are relevant 
for the face recognition can be determined by calculation of the corresponding spatial 

25 frequencies. Suitable threshold values for the expected value and the variance of the 
relevant spectral portions can be derived from reference image data. By these threshold 
values the grade of the sharpness of the image with respect to the problem of "face rec- 
ognition" can be determined. 

30 The global brightness of the image is given by the mean value of the gray values of the 
image. By comparison of the average gray value with an upper and a lower threshold it 
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can be determined whether the brightness of a portrait lies within a predetermined toler- 
ance. 

The contrast of an image is given by the ratio of brightness levels of neighbored area 
5 elements of an iniage. For verifyjng whether an image shows a sufficjent contract in a 
simplified way the variance of the image or of regions of the image, respectively, or the 
histogram of the entire image or of one or more sub-regions can be determined. A lower 
threshold for a sufficient contrast can be determined by a statistical analysis of an appro- 
priate image data set. 

10 

Highlights, i.e. reflection of the surface of an object due to an excessive Illumination or a 
spot light, lead to local atypical irradiation. Irradiation can be detected in the Image when 
global saturation values are reached or by local brightness discontinuities. Besides the 
brightness analysis advantageously area and shape measurements of the irradiated 
15 area(s) are performed in order to minimize errors. 

To verify the color distribution and color balance the available color Infomiation, which is 
usually present In an RGB representation, Is first of all transformed to a suitable color 
space (e.g., HSV: "Hue-Saturation-Value", or HSI: "Hue-Saturation-Intensity"). Then, the 
20 statistics of the image data are evaluated. Conveniently, the infomiation In the area of the 
face only should be used for this, since the color and the structure of the background are 
usually unknown. However, the case in that a background is specified and essentially the 
same over the entire data range is to be regarded as an exception. 

25 After evaluation of the statistics the results can be compared with model parameters. 
Comparison Is carried out either directly, e.g., by comparing histograms (e.g., minimizing 
the cross entropy) or by descriptive parameters (In the simplest case, e.g., by determining 
the mean value and variance and comparing with the corresponding model parameters; 
where appropriate some transformation of the distributions can follow in order to correct 

30 for deviations). 
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staining effects or other perturbations may influence the image quality in different ways. 
Staining spread over broad areas may, e.g., lead to a color balance that cannot be cor- 
rected. Smaller perturbations are detected by an analysis of the residuals between the 
original and the median-filtered image. Alternatively or in addition, morphological opera- 
5 tors can be used. 

The modulation G of a film or a camera to a light intensity I and thereby to a gray value 
is usually not linear but obeys a power function of the fomi G = F . By changing the 
gamma value the intensity distribution of the image can be transformed non-linearly. Val- 
10 ues for Y can be set either manually and after visual inspection or automatically. One ex- 
ample for a gamma correction is the correction of the well-known characteristic curves of 
several image processing and display hardware devices (e.g., monitor gamma, scanner 
gamma). 

15 In the following block 3 of the flow chart shown in Fig. 1 is described. 

It is the object of the segmentation to separate the individual objects of a scene from each 
other and from the background. For this, common methods for the segmentation of ob- 
jects can be used, however, advantageously specific methods for the segmentation of 
20 faces may be employed. 

In the common methods different approaches have been approved as, e.g., thresholding, 
multi spectral thresholding, but also pixel classification and optimization methods that aim 
at segmentation by relaxation. 

25 

Thresholding, e.g., is based on the fact that objects differ from the background by their 
gray values and is thus, employed almost exclusively in methods with controlled bound- 
ary conditions, in particular, with controlled illumination. For each pixel comparison with a 
threshold is performed and depending on the result of the comparison it is labeled as an 
30 object or background pixel. 
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In the specific methods for detection and segmentation effaces multi-step approaches 
are often used. 

Prior to the detection a "de-rotation" of the suspected face area may be attempted. In this, 
5 onestarts from tjie premise that a substantially frontal face is symmetric with respect to 
the nose. Determination of the symmetry axis, e.g., by a cepstrum analysis, and subse- 
quent de-rotation usually provide an area that possibly contains an upright face. 

There have been proposed and used different methods for face detection. Details can be 
10 found in "Neural Network-Based Face Detection". Rowley et al., IEEE Trans, on Pattern 
Analysis and Machine Intelligence". Vol. 20, No. 1, pp. 23-38, 1998, and "Face Detection 
in Color Images", Jain et al., IEEE Trans, on Pattern Analysis and Machine Intelligence, 
Vol. 24, No. 5, pp. 696-706, 2002. Faces can be segmented, e.g., with the help of the 
establishment of thresholds, in a suitable color space. Other approaches, e.g., by Rowley, 
15 are based on the classification of a pyramidal transformation of the image area by neural 
classificators that have been trained by exemplary image data. A preferred method for the 
segmentation of a face employs the above-mentioned graph matching method or some 
supplementary version of this method (e.g., HGM). 

20 Verifying features within the face region/background, which still is part of block 3, com- . 
prises: 

- method for determining the positions of the eyes 

- method for verifying the position of the head/face within the image (distance of the 
25 eyes to each other, distance to the edge of the image) 

- method for detemriining the size of the head 

- method for verifying and analyzing the head pose (frontoparallel rotation; depth ro- 
tation/nodding movements) 

30 In the following the individual methods are described in more detail. 

Determining the positions of the eyes: 

14 



The positions of the eyes are usually of particular interest for evaluating the observation 
of a predetermined specification for an inriage. Determining the positions of the eyes is a 
problem that is known in image processing for along time and solution approaches of dif- 
5.. ferent cpmplexitie^^ 

Jain et al. (see above for reference) propose a method according to which the eyes are 
detected by the distribution of chrominance and luminance within the face based on an 
initial detection of the face. The original color distributions are subject to morphological 
10 operations and are subsequently detected by thresholding on an image pyramid. 

Other methods are based on methods for comparing patterns either implicitly by neural 
classificators or explicitly by modeling and comparing the geometric features (e.g., by 
detecting the circles of the border of the pupil, the bows of the eyebrows, etc.). 

15 

Verifying the positions of the eyes within the image: 

For a favorable positioning of the face within the image the distances of the eyes from the 
edges of the image can be used alternatively. The distances of the left eye from the upper 
20 and the lower edges of the image are detected in terms of pixels and compared with pre- 
determined thresholds. Similar measurements are performed for the right eye. By empiri- 
cal investigations threshold values of about 40 pixels for the distance of each eye from 
the corresponding edge of the image and of about 80 pixels for the distances of the eye 
axis from the lower and upper edge of the image have been approved. 

25 

Verifying the distance of the eyes to each other: 

Subsequent to the detection of the positions of the eyes the distance between these posi- 
tions is detected and compared with a predetermined threshold. A distance of at least 50 
30 pixels between the centers of the eyes have been proven to be advantageous for the 
identification of persons. 
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Verifying the frontoparallel rotation: 

Verifying the frontoparallel rotation of a portrait is performed by calculating the rotation 
angle of the face with respect to a normal axis. This rotation angle can be calculated for 
5 known positions of the centers of the eyes by detecting the projected distance of the eyes 
in the x- and y-directions and by employing respective trigonometric relations. Since an 
accurate detection of the centers of the eyes is often problematic, e.g., due to image per- 
turbations, alternatively or in addition, the following more stable method can be per- 
formed: 

10 

1. Generating a translational invariant representation of the image that, however, still in- 
cludes rotational information. This is, e.g., given by the absolute value of the Fourier 
transform of the image. 

15 2. Generating a translational invariant representation of the mirrored image. 

3. Measuring the displacement of the rotation angle in this representation, e.g., by a 
transformation to polar coordinates and measurement of the displacement along the an- 
gle, e.g., by methods based on correlations. 

20 

For verifying the image quality the calculated rotation angle is compared with predeter- 
mined thresholds. Where appropriate the image can be rotated by a corresponding linear 
image operation such that the resulting image is no longer rotated or lies within predeter- 
mined thresholds. 

25 

Verifying the depth rotation: 

The relative rotation in space (fundamental matrix/essential matrix) can be determined 
from a set of point correspondences (of 8 or more pairs of points) between the image and 
30 a model with the help of a method that Is known in the literature as the 8-point-algorithm. 
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References for the latter mentioned methods are: "Complete Dense Estimation using 
Level Set Methods", O. Faugeras and A. Keriven, European Conference on Computer 
Vision, 1998; and "The Fundamental Matrix: Theory, Algorithms and Stability Analysis". 
Q. T. Luong and O. Faugeras, 1995. 

5 

The compensation of the depth rotation may be performed, e.g., by adaptation of the pa- 
rameters of a 3D head model on which the color or gray values of the given face, respec- 
tively, are projected. The thus calculated 3D model of the image is subsequently de- 
rotated, preferably to a normalized view, by the previously determined rotation parame- 
10 ters. 

In the case of the depth rotation also the calculated rotation angles are compared with 
predetermined thresholds for verifying the image quality. Where appropriate the image 
can be transformed by the above-described method such that the resulting image is no 
15 longer rotated or lies within the predetermined thresholds again. 

Besides the optimum image of the head/face area the background is also of relevance for 
optimized image recognition. Therefore, passport photographs typically have to obey the 
conditions of a unifomi, homogenously colored background. Subsequent to object local- 

20 ization and object segmentation (i.e. of the head or of the face) the background ("residual 
image") can be analyzed by the already described methods with respect to brightness 
(average brightness, variance), contrast, color and texture. If typical threshold values are 
exceeded, unsuitable images, i.e. images showing backgrounds that are considered as 
being inhomogenous or dominant by a viewer, can be rejected or can be subject to a cor- 

25 rection method for replacing the background by a more suitable one by means of elec- 
tronic image manipulation. For a pixel-by-pixel replacement backgrounds showing con- 
stant brightness and colors can advantageously be chosen. 

If the object background that is visible in the image does not fulfill the verification criteria, 
30 this background can be replaced by another one that observes the verification criteria by 
methods of image manipulation. This is realized by overwriting pixel-by-pixel the area that 
has been marked as image background area during the segmentation process at which. 
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in the simplest case, a constant color and brightness are assigned to each pixel. The se- 
lection of a suitable background color and brightness may advantageously be carried out 
by also taking into account the considered object itself in order to optimize the difference 
in contrast between the object and the background. 

5 

In the following block 4, i.e. verifying local criteria of the head/face region, is described. 

First, the verification of the image area with respect to completeness, e.g., verification 
whether both eyes, the nose, the mouth portion are visible, whether the edges of the face 
10 surpass the image, (segmentation completely within the image) has to be mentioned. 

To verify whether all of the substantial parts of the face are visible a method based on, 
e.g., a graph matching method (e.g., HGM), may be preferred. Subsequent to the match- 
ing and placing of the knots based on the overall evaluation of the graph and thus, not 
15 significantly influenced by perturbations similarities of local knots or groups of knots can 
be evaluated. This is done by either verifying whether the similarities to model knots (e.g., 
in form of examples) are sufficiently high or explicitly by classifying in the considered posi- 
tive case and the corresponding negative case (e.g., similarity to "normal" opened eye, 
closed eye, eye patch). 

20 

In addition, it can be verified by comparison of the maxima and minima of the coordinates 
of the positioned graph with the dimensions of the image whether (a) the face is com- 
pletely present in the image and (b) the size and position of the face observe quality 
standards within predetermined tolerable deviations. 

25 

in specific cases it can be useful to detect worn eyeglasses separately. For example, it 
might be possible that the quality standards only allow for images without eyeglasses 
only. 

30 A possible method for this is the method of bunch graph matching with annotated graphs 
first described by L. Wiskott: "Face recognition and gender determination", L. Wiskott et 
al.. International Workshop on Automatic Face-and-Gesture-Recognition, pp. 92-97, 
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Zurich, 26.-28. 06. 1995. According to this method subsequent to the positioning of the 
graph it is verified which knot of the graph stack resembles most the knot of the current 
graph. If the graph stack has been previously annotated, e.g., in the way that the corre- 
sponding image shows a person wearing eyeglasses, this annotation is also assigned to 
5 the current knot. By a final evaluation of the annotations of all current knots (e.g., by ma- 
jority vote) one can judge that eyeglasses are worn or not in the image. 

Of particular importance is the verification - and where appropriate the manipulation - of 
the region of the eyes. 

10 

An approach for discriminating between closed and opened eyes has already been 
sketched above. Subsequent to the positioning of the graph it is, in particular, readily 
possible to recognize the "red eye" problem that occurs due to the usage of flash lights by 
evaluation of the color information at the positions of the eye knots. Moreover, correction 
15 by a modification of the color information in the corresponding regions is feasable. 

For the verification of the mouth region substantially the same holds as what was said 
about the region of the eyes. In addition, however, there evaluating the deviations of the 
current knot positions from the positions of corresponding knots of reference models is 
20 feasable, e.g., by using the sum of the absolute values of all positional deviations with 
respect to a reference knot in the case of an opened or closed mouth, respectively. In the 
simplest case, the decision can be drawn by a comparison with some threshold. 

Verification of the presence of a headgear can be performed according to the following 
25 pragmatic approach: Subsequent to the successful positioning of the graph the knots that 
describe the upper head edge can be evaluated. Verification with respect to color and 
shape can be preferred. The first can be done by analysis of the original image, the latter 
by evaluating the similarity with model images. If significant deviations are present in one 
of the both cases, some headgear might be present. Such images should be subject to 
30 visual inspection by a human operator (-> block 6). 
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In the following reference is made to block 5 (image correction), in so far as the corre- 
sponding methods have not previously been used and described in one of the preceding 
blocks 2 to 4. 

5 In particular, methods for contrast, brightness and color correction in the head/face area 
are to be mentioned. In addition to already mentioned correction methods ("manipulation 
methods") here, corrections are not carried out globally but rather restricted to the 
head/face area. The above-mentioned adaptation methods based on histograms or color 
space transformations are accordingly to be applied to these areas obtained by segmen- 
10 tation. 

Finally, reference is made to block 6 of Figure 1 . 

Subsequent to performance of the above-mentioned methods it has to be signalized to a 
15 user whether the image (or where appropriate the corrected image) that is subject to the 
verification method observes the predetermined quality standards. If so, the image (or 
where appropriate the corrected image), e.g., can be stored to a database, transmitted on 
an identity document, etc. 

20 The method exemplified with the help of the Figure serves for illustration purposes of the 
principles on which the invention is based only. The scope of protection of the invention is 
defined by the appended claims. 
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